Inpainting device and method using segmentation of reference region

ABSTRACT

An inpainting device and method using the segmentation of a reference region are disclosed. An inpainting device using the segmentation of a reference region includes a region determination unit configured to determine an inpainting target region and a reference region within video image content, an inpainting group setting unit configured to set inpainting groups using pixel values within the reference region; an inpainting unit configured to perform segmentation inpainting adapted to perform inpainting on a segmented target region using the segmented reference region on an inpainting group basis and non-segmentation inpainting adapted to perform inpainting on the target region using the reference region, and an information provision unit configured to visually provide the result of the segmentation inpainting, the result of the non-segmentation inpainting, and information about the difference between the results of the segmentation inpainting and the non-segmentation inpainting to the user.

CROSS REFERENCE TO RELATED APPLICATION

The present invention claims the benefits of the filing date of KoreanPatent Application No. 10-2014-0009197 filed on Jan. 24, 2014, in theKorean Patent and Trademark Office. Further, this application is theNational Phase application of International Application No.PCT/KR2014/009924 filed on Oct. 22, 2014, the content of which isincorporated herein in its entirety.

TECHNICAL FIELD

The present invention relates to an inpainting device and method usingthe segmentation of a reference region, and more particularly to aninpainting device and method using the segmentation of a referenceregion that can perform inpainting without segmenting an inpaintingregion and after segmenting an inpainting region in multiple stages andthat can provide information about the difference between the results ofrespective types of inpainting to the user.

BACKGROUND ART

With the development of digital signal processing technology, virtualadvertising that inserts a virtual image into a video has beenpopularized, and indirect advertising has established itself as aneffective advertising means. Accordingly, interest has been rapidlyincreasing in virtual indirect advertising that combines a virtualadvertisement with an indirect advertisement and naturally inserts anadvertisement image into a previously generated video.

In connection with virtual indirect advertising, inpainting technologythat naturally inserts a virtual image into a previously generated videoor that eliminates part of an image of a previously generated video andperforms natural restoration to achieve harmonization with a backgroundis fundamental.

In inpainting technology, a reference region to be used for restorationis required to restore a target region. When a reference region isdesignated in a non-limiting manner, as in a case where the referenceregion is designated as an overall video frame, a problem arises in thatcomputational load rapidly increases. When overall inpainting isperformed regardless of the correction between a reference region and aregion to be restored even when the reference region is limited, aproblem arises in that an unintentional inpainting result occurs.

Accordingly, there is a growing need for new inpainting technology thatperforms inpainting without segmenting an inpainting region and aftersegmenting an inpainting region in multiple stages and that providesinformation about the difference between the results of respective typesof inpainting to the user, thereby overcoming a problem in which anawkward inpainting result occurs when inpainting is performed in asingle mode.

Related preceding technology includes Korean Patent ApplicationPublication No. 10-2013-0056407 published on May 30, 2013 (entitled“Inpainting System and Method for H.264 for Error Concealment Image”).

DISCLOSURE Technical Problem

An object of the present invention is to perform inpainting withoutsegmenting an inpainting region and after segmenting an inpaintingregion in multiple stages and provide information about the differencebetween the results of respective types of inpainting to the user,thereby overcoming a problem in which an awkward inpainting resultoccurs when inpainting is performed in a single mode.

Furthermore, an object of the present invention is to provide varioustypes of inpainting result information to a user, receive feedback fromthe user, and incorporate the feedback into a final inpainting result,thereby performing more natural inpainting.

Technical Solution

In order to accomplish the above objects, the present invention providesan inpainting device using the segmentation of a reference region,including: a region determination unit configured to determine aninpainting target region present within video image content and areference region composed of pixels present within a preset distancefrom the boundary pixels of the inpainting target region; an inpaintinggroup setting unit configured to set inpainting groups composed ofrespective pairs of segmented reference regions obtained by segmentingthe reference region using pixel values within the reference region andsegmented target regions obtained by segmenting the target region so asto correspond to the segmented reference region; an inpainting unitconfigured to perform segmentation inpainting adapted to performinpainting on a segmented target region using the segmented referenceregion on an inpainting group basis, and non-segmentation inpaintingadapted to perform inpainting on the target region using the referenceregion; and an information provision unit configured to visually providethe result of the segmentation inpainting, the result of thenon-segmentation inpainting, and information about the differencebetween the results of the segmentation inpainting and thenon-segmentation inpainting to the user.

In this case, the inpainting group setting unit may further include agroup set generation unit configured to differently generate group setsbased on the number of inpainting groups; and the group set generationunit may generate a first group set in which the number of inpaintinggroups has been set to the first number of segmentations and generate asecond group set in which the number of inpainting groups has been setto the second number of segmentations different from the first number ofsegmentations.

In this case, the segmentation inpainting may include first segmentationinpainting adapted to be performed on the first group set and secondsegmentation inpainting adapted to be performed on the second group set.

In this case, the result of the segmentation inpainting may include theresults of the first segmentation inpainting and the second segmentationinpainting.

In this case, the information provision unit may further visuallydisplay information about the difference between the results of thefirst segmentation inpainting and the second segmentation inpainting.

In this case, the group set generation unit additionally may generatethe second group set in which the number of inpainting groups is equalto or smaller than the reference number of groups when the number ofinpainting groups of the first group set exceeds the preset referencenumber of groups.

In this case, the group set generation unit may generate the secondsegmentation set by merging any two or more of the inpainting groups ofthe first group set when the first number of segmentations exceeds thepreset reference number of segmentations.

In this case, the inpainting group setting unit may calculate the edgevalues of pixels within the reference region, may determine whether avariation in edge values is below a preset reference variation, and maygroup the pixels within the reference region and set the grouped pixelsas the segmented reference region based on the result of thedetermination.

In this case, the inpainting group setting unit may determine whetherthe variation in edge values is below the preset reference variation,and may group the pixels within the reference region and set the groupedpixels as the segmented reference region based on the result of thedetermination.

In this case, the inpainting group setting unit may determine whetherthe pixel values within the reference region fall within a preset rangeof values similar to a reference pixel value, and may set pixels fallingwithin the similar range as an identical inpainting group.

In this case, the inpainting group setting unit may segment the targetregion using the pattern of the segmentation boundary line of thereference region.

Furthermore, the present invention provides an inpainting method usingthe segmentation of a reference region, including: determining aninpainting target region present within video image content and areference region composed of pixels present within a preset distancefrom the boundary pixels of the inpainting target region; settinginpainting groups composed of respective pairs of segmented referenceregions obtained by segmenting the reference region using pixel valueswithin the reference region and segmented target regions obtained bysegmenting the target region so as to correspond to the segmentedreference region; performing segmentation inpainting adapted to performinpainting on a segmented target region using the segmented referenceregion on an inpainting group basis and non-segmentation inpaintingadapted to perform inpainting on the target region using the referenceregion; and visually providing the result of the segmentationinpainting, the result of the non-segmentation inpainting, andinformation about the difference between the results of the segmentationinpainting and the non-segmentation inpainting to the user.

In this case, setting the inpainting groups may further includedifferently generating group sets based on the number of inpaintinggroups; generating the group sets may include generating a first groupset in which the number of inpainting groups has been set to the firstnumber of segmentations and generating a second group set in which thenumber of inpainting groups has been set to the second number ofsegmentations different from the first number of segmentations.

In this case, the segmentation inpainting may include first segmentationinpainting adapted to be performed on the first group set and secondsegmentation inpainting adapted to be performed on the second group set.

In this case, the result of the segmentation inpainting may include theresults of the first segmentation inpainting and the second segmentationinpainting.

In this case, providing the information about the difference may includefurther visually displaying information about the difference between theresults of the first segmentation inpainting and the second segmentationinpainting.

In this case, generating the group sets may include additionallygenerating the second group set in which the number of inpainting groupsis equal to or smaller than the reference number of groups when thenumber of inpainting groups of the first group set exceeds the presetreference number of groups.

In this case, generating the group sets may include generating thesecond segmentation set by merging any two or more of the inpaintinggroups of the first group set when the first number of segmentationsexceeds the preset reference number of segmentations.

Advantageous Effects

According to the present invention, inpainting is performed withoutsegmenting an inpainting region and after segmenting an inpaintingregion in multiple stages, and information about the difference betweenthe results of respective types of inpainting is provided to the user,thereby overcoming a problem in which an awkward inpainting resultoccurs when inpainting is performed in a single mode.

Furthermore, according to the present invention, various types ofinpainting result information are provided to a user, feedback isreceived from the user, and the feedback is incorporated into a finalinpainting result, thereby performing more natural inpainting.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an inpainting device using thesegmentation of a reference region according to an embodiment of thepresent invention;

FIG. 2 is a diagram showing an example of the step of determining atarget region and a reference region according to the present invention;

FIGS. 3 to 6 are diagrams showing an example of the step of settinginpainting groups according to the present invention;

FIGS. 7 and 8 are diagrams showing an example of the step of generatinggroup sets according to the present invention; and

FIG. 9 is an operation flowchart showing an inpainting method using thesegmentation of a reference region according to an embodiment of thepresent invention.

MODE FOR INVENTION

Preferred embodiments of the present invention will be described indetail below with reference to the accompanying drawings. However,detailed descriptions of well-known functions and configurations thatmay make the gist of the present invention unnecessarily obscure will beomitted in the following description and the attached drawings.Furthermore, it should be noted that the same components are designatedby the same reference symbols throughout the drawings as much aspossible.

The terms and words used in the following specification and the claimsshould not be restrictively interpreted as having common and dictionarymeanings, but should be interpreted as having meanings and concepts inconformity with the technical spirit of the present invention based onthe principle that an inventor can appropriately define the concepts ofterms in order to describe his or her invention in the best way.Accordingly, since the embodiments described in the presentspecification and the configurations shown in the drawings merelycorrespond to the preferred embodiments of the present invention and donot cover all the technical spirit of the present invention, it shouldbe appreciated that there may be various equivalents and modificationsthat may replace the configurations at the time at which the presentapplication is filed. Furthermore, the terms “first,” “second” and so onare used to describe various components, and are intended to distinguishone component from other components but are not intended to limit thesecomponents.

An inpainting device using the segmentation of a reference regionaccording to an embodiment of the present invention is described below.

FIG. 1 is a block diagram showing an inpainting device using thesegmentation of a reference region according to an embodiment of thepresent invention.

Referring to FIG. 1, the inpainting device using the segmentation of areference region according to the embodiment of the present inventionincludes a region determination unit 110, an inpainting group settingunit 120, an inpainting unit 130, and an information provision unit 140.

The region determination unit 110 determines an inpainting target regionpresent within video image content and a reference region composed ofpixels present within a preset distance from the boundary pixels of theinpainting target region.

In this case, the inpainting target region refers to a region on whichinpainting will be performed within the video image content.

In this case, the inpainting target region may be determined by theinput of a user.

The inpainting group setting unit 120 sets inpainting groups composed ofrespective pairs of segmented reference regions obtained by segmentingthe reference region using pixel values within the reference region andsegmented target regions obtained by segmenting the target region so asto correspond to the segmented reference region.

In this case, the inpainting group setting unit 120 may calculate theedge values of pixels within the reference region, may determine whethera variation in edge values is below a preset reference variation, andmay group the pixels within the reference region and set the groupedpixels as the segmented reference region based on the result of thedetermination.

In this case, the inpainting group setting unit 120 may determinewhether the variation in edge values is below the preset referencevariation, and may group the pixels within the reference region and setthe grouped pixels as the segmented reference region based on the resultof the determination.

In this case, the reference variation may be a reference value that ispreset so as to determine pixels having large variations in edge valuesamong the pixels within the reference region.

In this case, the inpainting group setting unit 120 may determinewhether the pixel values within the reference region fall within apreset range of values similar to a reference pixel value, and may setpixels falling within the similar range as the same inpainting group.

In this case, the reference pixel value may be a reference value that ispreset so as to determine pixels having similar pixel values among thepixels within the reference region.

In this case, the reference pixel value may be a reference value that ispreset to a pixel value having a predetermined or higher distributionusing a distribution histogram of pixel values.

In this case, the inpainting group setting unit 120 may segment thetarget region using the pattern of the segmentation boundary line of thereference region.

For example, the inpainting group setting unit 120 may segment thetarget region into rectilinear patterns when the segmentation boundaryline of the reference region is a rectilinear pattern, and may alsosegment the target region into zigzag patterns when the segmentationboundary line of the reference region is a zigzag pattern.

In this case, the inpainting group setting unit 120 may include a groupset generation unit configured to differently generate group sets basedon the number of inpainting groups.

In this case, the group set is a set in which a method of segmenting thesame target region and reference region in various manners has been set.

In this case, the group set generation unit may generate a first groupset in which the number of inpainting groups has been set to the firstnumber of segmentations, and may generate a second group set in whichthe number of inpainting groups has been set to the second number ofsegmentations different from the first number of segmentations.

For example, the group set generation unit may generate a first groupset in which the number of inpainting groups has been set to 2segmentations and a second group set in which the number of inpaintinggroups has been set to 3 segmentations.

In this case, the group set generation unit may additionally generate asecond group set in which the number of inpainting groups is equal to orsmaller than the reference number of groups when the number ofinpainting groups of the first group set exceeds the preset referencenumber of groups.

In this case, the reference number of groups may be the upper limitvalue of the number of inpainting groups that is preset so as to limitsegmentation into an excessively large number of inpainting groups.

For example, the group set generation unit may additionally generates asecond group set in which the number of inpainting groups is equal to orsmaller than 6 when the number of inpainting groups of the first groupset exceeds 6, i.e., the preset reference number of groups.

In this case, the group set generation unit may generate the secondsegmentation set by merging any two or more of the inpainting groups ofthe first group set when the first number of segmentations exceeds thepreset reference number of segmentations.

For example, the group set generation unit may generate a second groupset having a total of 6 inpainting groups by merging any two of theinpainting groups of the first group set when the reference number ofgroups is 6 and the number of inpainting groups of the first group setis 7.

In this case, the group set generation unit may merge groups in whichthe representative pixel values of respective inpainting groups aresimilar.

The inpainting unit 130 performs segmentation inpainting adapted toperform inpainting on a segmented target region using the segmentedreference region on an inpainting group basis, and non-segmentationinpainting adapted to perform inpainting on a target region using thereference region.

In this case, the non-segmentation inpainting is inpainting that isperformed without segmenting the reference region and the target region.

In this case, the segmentation inpainting may include first segmentationinpainting adapted to be performed on the first group set, and secondsegmentation inpainting adapted to be performed on the second group set.

In this case, the result of the segmentation inpainting may include theresult of the first segmentation inpainting and the result of the secondsegmentation inpainting.

The information provision unit 140 visually provides the result of thesegmentation inpainting, the result of the non-segmentation inpainting,and information about the difference between the results of thesegmentation inpainting and the non-segmentation inpainting to the user.

In this case, the information provision unit may further visuallydisplay information about the difference between the results of thefirst segmentation inpainting and the second segmentation inpainting.

As described above, the inpainting device using the segmentation of areference region according to the embodiment of the present inventionmay perform inpainting without segmenting the inpainting region andafter segmenting the inpainting region in multiple stages, and mayprovide information about the difference between the results ofrespective types of inpainting to the user. Furthermore, more naturalinpainting can be performed by providing various types of inpaintingresult information to the user, receiving feedback from the user andincorporating the feedback into a final inpainting result.

The determination of regions according to an embodiment of the presentinvention is described below.

FIG. 2 is a diagram showing an example of the step of determining atarget region and a reference region according to the present invention.

Referring to FIG. 2, at the step of determining a target region and areference region according to the present invention according to thepresent invention, a region of pixels corresponding to a target on whichinpainting will be performed within a video image frame 200 isdetermined, and a region of pixels which are referred to duringinpainting.

It can be seen that the pixels corresponding to a target on whichinpainting will be performed are pixels within a target region andpixels inside a target region boundary line 210.

It can be seen that the pixels which are referred to during inpaintingare pixels between the target region boundary line 210 and a referenceregion boundary line 220 that is a preset distance away from the targetregion boundary line 210 in an outward direction.

In this case, the inpainting target region or target region boundaryline 210 may be determined by the input of the user.

The setting of inpainting groups according to an embodiment of thepresent invention is described below.

FIGS. 3 to 6 are diagrams showing an example of the step of settinginpainting groups according to the present invention.

FIGS. 3 to 6 show only the target region and reference region of FIG. 2.Inpainting groups are sets in a sequence from FIG. 3 to FIG. 6.

Referring to FIG. 3, it can be seen that the step of setting inpaintinggroups according to the present invention is directed to only the targetregion 310 and the reference region 320.

Referring to FIG. 4, at the step of setting inpainting groups accordingto the present invention, the reference region 320 is segmented usingpixel values within the reference region 320.

That is, at the step of setting inpainting groups according to thepresent invention, a region corresponding to a portion outside a desk inFIG. 2 is set as a first segmented reference region 421, and a regioncorresponding to a portion inside the desk is set as a second segmentedreference region 422 using pixel values within the reference region 320.

In this case, at the step of setting inpainting groups according to thepresent invention, it may be determined whether the pixel values withinthe reference region 320 fall within a preset range of values similar toa reference pixel value, and pixels falling within the range of similarvalues may be set as the same segmented reference region.

In this case, the reference pixel value may be a reference value that ispreset so as to determine pixels having similar pixel values amongpixels within the reference region.

For example, when the pixels corresponding to the portion outside thedesk in FIG. 4 fall within a range of values similar to a referencepixel value for the portion outside the desk, a region corresponding tothe portion outside the desk may be set as the first segmented referenceregion 421.

In this case, the reference pixel value may be a reference value that ispreset to a pixel value having a predetermined or higher distributionusing a distribution histogram of pixel values.

For example, in FIG. 4, when the value that is preset to a pixel valuehaving a predetermined or higher distribution is determined to be thereference pixel value using a distribution histogram of pixel values,the reference pixel value may include any one of the pixel values of theportion outside the desk and any one of the pixel values of the portioninside the desk.

In this case, at the step of setting inpainting groups according to thepresent invention, the edge values of the pixels within the referenceregion 320 may be calculated, it may be determined whether a variationin edge values is below a preset reference variation, and the referenceregion may be segmented based on the result of the determination.

In this case, the reference variation may be a reference value that ispreset so as to determine pixels having large variations in edge valuesamong the pixels within the reference region.

For example, in FIG. 4, when the edge values of the boundary lines ofthe portion outside the desk and a portion below the desk are above thereference variation, a region corresponding to the portion outside thedesk may be set as the first segmented reference region 421, and aregion corresponding to the portion inside the desk may be set as thesecond segmented reference region 422.

Referring to FIG. 5, at the step of setting inpainting groups accordingto the present invention, the target region 310 is segmented so as tocorrespond to the segmented reference regions 421 and 422.

In this case, at the step of setting inpainting groups according to thepresent invention, the target region 310 may be segmented using thepattern of a boundary line between the segmented reference regions 421and 422.

For example, at the step of setting inpainting groups according to thepresent invention, when the boundary line between the segmentedreference regions 421 and 422 is a rectilinear pattern, the targetregion 310 may be segmented into segmented target regions 511 and 512having a boundary line based on the rectilinear pattern.

That is, at the step of setting inpainting groups according to thepresent invention, when a boundary line between the segmented referenceregions 421 and 422 has a zigzag pattern, the target region 310 may besegmented into the segmented target regions 511 and 512 having aboundary line based on the zigzag pattern.

Referring to FIG. 6, at the step of setting inpainting groups accordingto the present invention, inpainting groups 601 and 602 composed ofrespective pairs of the segmented reference regions 421 and 422 and thesegmented target regions 511 and 512 corresponding to the segmentedreference regions 421 and 422, respectively, are set.

For example, at the step of setting inpainting groups according to thepresent invention, when the segmented reference region 421 and segmentedtarget region 511 corresponding to each other, the inpainting group 601composed of a pair of the segmented reference region 421 and segmentedtarget region 511 may be set.

That is, as described in the series of descriptions of FIGS. 3 to 6, atthe step of setting inpainting groups according to the presentinvention, it can be seen that the reference region 320 is segmented,the target region 310 is segmented accordingly, and the inpaintinggroups 601 and 602 of respective pairs of the segmented reference region421 and 422 and the segmented target region 511 and 512 corresponding tothe segmented reference region 421 and 422, respectively, are set.

The generation of group sets according to an embodiment of the presentinvention is described below.

FIGS. 7 and 8 are diagrams showing an example of the step of generatinggroup sets according to the present invention.

FIG. 7 shows a first group set in which a total of 2 inpainting groupshave been set, and FIG. 8 shows a second group set in which a total of 4inpainting groups have been set.

At the step of generating group sets according to the present invention,group sets may be differently generated based on the number ofinpainting groups.

In this case, the group set is a set in which a method of segmenting thesame target region and reference region in various manners has been set.

Referring to FIG. 7, it can be seen that a first inpainting group 701corresponding to a portion outside a desk and a second inpainting group702 corresponding to a portion inside the desk have been set as a firstgroup set.

Referring to FIG. 8, it can be seen that inpainting groups 801 to 803based on the colors of a background in a portion outside a desk and afourth inpainting group 804 in a portion inside the desk have been setas a second group set.

As described above, it can be seen that, at the step of generating groupsets according to the present invention, various inpainting groupsettings that are generated when the reference value, such as a similarpixel value range, varies at the step of setting inpainting groups maybe generated as respective group sets.

An inpainting method using the segmentation of a reference regionaccording to an embodiment of the present invention is described below.

FIG. 9 is an operation flowchart showing an inpainting method using thesegmentation of a reference region according to an embodiment of thepresent invention.

Referring to FIG. 9, in the inpainting method using the segmentation ofa reference region according to the embodiment of the present invention,an inpainting target region present within video image content and areference region composed of pixels present within a preset distancefrom the boundary pixels of the inpainting target region are determinedat step S910.

In this case, the inpainting target region refers to a region on whichinpainting will be performed within the video image content.

In this case, the inpainting target region may be determined by theinput of a user.

Furthermore, in the inpainting method using the segmentation of areference region according to the embodiment of the present invention,inpainting groups composed of respective pairs of segmented referenceregions obtained by segmenting the reference region using pixel valueswithin the reference region and segmented target regions obtained bysegmenting the target region so as to correspond to the segmentedreference region are set at step S920.

In this case, at step S920, the edge values of pixels within thereference region may be calculated, it may be determined whether avariation in edge values is below a preset reference variation, and thepixels within the reference region may be grouped and set as thesegmented reference region based on the result of the determination.

In this case, at step S920, it may be determined whether the variationin edge values is below the preset reference variation, and the pixelswithin the reference region may be grouped and set as the segmentedreference region based on the result of the determination.

In this case, the reference variation may be a reference value that ispreset so as to determine pixels having large variations in edge valuesamong the pixels within the reference region.

In this case, at step S920, it may be determined whether the pixelvalues within the reference region fall within a preset range of valuessimilar to a reference pixel value, and pixels falling within thesimilar range may be set as the same inpainting group.

In this case, the reference pixel value may be a reference value that ispreset so as to determine pixels having similar pixel values among thepixels within the reference region.

In this case, the reference pixel value may be a reference value that ispreset to a pixel value having a predetermined or higher distributionusing a distribution histogram of pixel values.

In this case, at step S920, the target region may be segmented using thepattern of the segmentation boundary line of the reference region.

For example, at step S920, the target region may be segmented intorectilinear patterns when the segmentation boundary line of thereference region is a rectilinear pattern, and the target region may bealso segmented into zigzag patterns when the segmentation boundary lineof the reference region is a zigzag pattern.

In this case, step S920 may further include the step of differentlygenerating group sets based on the number of inpainting groups.

In this case, the group set is a set in which a method of segmenting thesame target region and reference region in various manners has been set.

In this case, at the step of generating group sets, a first group set inwhich the number of inpainting groups has been set to the first numberof segmentations may be generated, and a second group set in which thenumber of inpainting groups has been set to the second number ofsegmentations different from the first number of segmentations may begenerated.

For example, at the step of generating group sets, a first group set inwhich the number of inpainting groups has been set to 2 segmentationsmay be generated, and a second group set in which the number ofinpainting groups has been set to 3 segmentations may be generated.

In this case, at the step of generating group sets, when the number ofinpainting groups of the first group set exceeds the preset referencenumber of groups, a second group set in which the number of inpaintinggroups is equal to or smaller than the reference number of groups may beadditionally generated.

In this case, the reference number of groups may be the upper limitvalue of the number of inpainting groups that is preset so as to limitsegmentation into an excessively large number of inpainting groups.

For example, at the step of generating group sets, when the number ofinpainting groups of the first group set exceeds 6, i.e., the presetreference number of groups, a second group set in which the number ofinpainting groups is equal to or smaller than 6 may be additionallygenerated.

In this case, at the step of generating group sets, when the firstnumber of segmentations exceeds the preset reference number ofsegmentations, the second segmentation set may be generated by mergingany two or more of the inpainting groups of the first group set.

For example, at the step of generating group sets, when the referencenumber of groups is 6 and the number of inpainting groups of the firstgroup set is 7, a second group set having a total of 6 inpainting groupsmay be generated by merging any two of the inpainting groups of thefirst group set.

In this case, at the step of generating group sets, groups in which therepresentative pixel values of respective inpainting groups are similarmay be merged.

Furthermore, in the inpainting method using the segmentation of areference region according to the embodiment of the present invention,segmentation inpainting adapted to perform inpainting on a segmentedtarget region using the segmented reference region on an inpaintinggroup basis is performed, and non-segmentation inpainting adapted toperform inpainting on a target region using the reference region isperformed at step S930.

In this case, the non-segmentation inpainting is inpainting that isperformed without segmenting the reference region and the target region.

In this case, the segmentation inpainting may include first segmentationinpainting adapted to be performed on the first group set, and secondsegmentation inpainting adapted to be performed on the second group set.

In this case, the result of the segmentation inpainting may include theresult of the first segmentation inpainting and the result of the secondsegmentation inpainting.

Furthermore, in the inpainting method using the segmentation of areference region according to the embodiment of the present invention,the result of the segmentation inpainting, the result of thenon-segmentation inpainting, and information about the differencebetween the results of the segmentation inpainting and thenon-segmentation inpainting are visually provided to the user.

In this case, at step S540, information about the difference between theresults of the first segmentation inpainting and the second segmentationinpainting may be further visually displayed.

As described above, in the inpainting method using the segmentation of areference region according to the embodiment of the present invention,inpainting may be performed without segmenting the inpainting region andafter segmenting the inpainting region in multiple stages, andinformation about the difference between the results of respective typesof inpainting may be provided to the user. Furthermore, more naturalinpainting can be performed by providing various types of inpaintingresult information to the user, receiving feedback from the user andincorporating the feedback into a final inpainting result.

The steps shown in FIG. 9 may be performed in the sequence shown in FIG.9, in a sequence reverse to the former sequence, or concurrently.

The inpainting method using the segmentation of a reference regionaccording to the present invention may be implemented as a program orsmart phone app that can be executed by various computer means. In thiscase, the program or smart phone app may be recorded on acomputer-readable storage medium. The computer-readable storage mediummay include program instructions, data files, and data structures solelyor in combination. Program instructions recorded on the storage mediummay have been specially designed and configured for the presentinvention, or may be known to or available to those who have ordinaryknowledge in the field of computer software. Examples of thecomputer-readable storage medium include all types of hardware devicesspecially configured to record and execute program instructions, such asmagnetic media, such as a hard disk, a floppy disk, and magnetic tape,optical media, such as compact disk (CD)-read only memory (ROM) and adigital versatile disk (DVD), magneto-optical media, such as a flopticaldisk, ROM, random access memory (RAM), and flash memory. Examples of theprogram instructions include machine code, such as code created by acompiler, and high-level language code executable by a computer using aninterpreter. These hardware devices may be configured to operate as oneor more software modules in order to perform the operation of thepresent invention, and the vice versa.

The inpainting device and method using the segmentation of a referenceregion according to the present invention are not limited to theconfigurations and methods of the above-described embodiments, but someor all of the embodiments may be configured to be selectively combinedsuch that the embodiments can be modified in various manners.

INDUSTRIAL APPLICABILITY

According to the present invention, inpainting is performed withoutsegmenting an inpainting region and after segmenting an inpaintingregion in multiple stages, and information about the difference betweenthe results of respective types of inpainting is provided to the user,thereby overcoming a problem in which an awkward inpainting resultoccurs when inpainting is performed in a single mode. In particular,when the trend in which interest in virtual indirect advertising thatincreases an advertising effect by inserting an image of an advertisingitem into a drama or sports broadcast has been increasing in line with arecent increase in the scale of the advertising market and the fact thatinpainting technology that naturally inserts a virtual image into apreviously generated video or that eliminates part of an image of apreviously generated video and performs natural restoration to achieveharmonization with a background is fundamental are taken intoconsideration, the utilization of the inpainting device and method usingthe segmentation of a reference region according to the presentinvention will be high.

The invention claimed is:
 1. An inpainting device using segmentation ofa reference region, comprising: a region determination unit configuredto determine an inpainting target region present within video imagecontent and a reference region composed of pixels present within apreset distance from boundary pixels of the inpainting target region; aninpainting group setting unit configured to set inpainting groupscomposed of respective pairs of segmented reference regions obtained bysegmenting the reference region using pixel values within the referenceregion and segmented target regions obtained by segmenting the targetregion so as to correspond to the segmented reference region; aninpainting unit configured to perform segmentation inpainting adapted toperform inpainting on a segmented target region using the segmentedreference region on an inpainting group basis, and non-segmentationinpainting adapted to perform inpainting on the target region using thereference region; and an information provision unit configured tovisually provide a result of the segmentation inpainting, a result ofthe non-segmentation inpainting, and information about a differencebetween the results of the segmentation inpainting and thenon-segmentation inpainting to the user, wherein the inpainting groupsetting unit further comprises a group set generation unit configured todifferently generate group sets based on a number of inpainting groups,and wherein the group set generation unit generates a first group set inwhich the number of inpainting groups has been set to a first number ofsegmentations, and generates a second group set in which the number ofinpainting groups has been set to a second number of segmentationsdifferent from the first number of segmentations.
 2. The inpaintingdevice of claim 1, wherein the segmentation inpainting comprises firstsegmentation inpainting adapted to be performed on the first group setand second segmentation inpainting adapted to be performed on the secondgroup set.
 3. The inpainting device of claim 2, wherein the result ofthe segmentation inpainting comprises results of the first segmentationinpainting and the second segmentation inpainting.
 4. The inpaintingdevice of claim 3, wherein the information provision unit furthervisually displays information about a difference between the results ofthe first segmentation inpainting and the second segmentationinpainting.
 5. The inpainting device of claim 4, wherein the group setgeneration unit additionally generates the second group set in which thenumber of inpainting groups is equal to or smaller than a referencenumber of groups when a number of inpainting groups of the first groupset exceeds the preset reference number of groups.
 6. The inpaintingdevice of claim 5, wherein the group set generation unit generates thesecond segmentation set by merging any two or more of the inpaintinggroups of the first group set when the first number of segmentationsexceeds a preset reference number of segmentations.
 7. The inpaintingdevice of claim 6, wherein the inpainting group setting unit calculatesedge values of pixels within the reference region, determines whether avariation in edge values is below a preset reference variation, andgroups the pixels within the reference region and sets the groupedpixels as the segmented reference region based on a result of thedetermination.
 8. The inpainting device of claim 7, wherein theinpainting group setting unit determines whether the variation in edgevalues is below the preset reference variation, and groups the pixelswithin the reference region and sets the grouped pixels as the segmentedreference region based on a result of the determination.
 9. Theinpainting device of claim 6, wherein the inpainting group setting unitdetermines whether the pixel values within the reference region fallwithin a preset range of values similar to a reference pixel value, andsets pixels falling within the similar range as an identical inpaintinggroup.
 10. The inpainting device of claim 6, wherein the inpaintinggroup setting unit segments the target region using a pattern of asegmentation boundary line of the reference region.
 11. An inpaintingmethod using segmentation of a reference region, comprising: determiningan inpainting target region present within video image content and areference region composed of pixels present within a preset distancefrom boundary pixels of the inpainting target region; setting inpaintinggroups composed of respective pairs of segmented reference regionsobtained by segmenting the reference region using pixel values withinthe reference region and segmented target regions obtained by segmentingthe target region so as to correspond to the segmented reference region;performing segmentation inpainting adapted to perform inpainting on asegmented target region using the segmented reference region on aninpainting group basis, and non-segmentation inpainting adapted toperform inpainting on the target region using the reference region; andvisually providing a result of the segmentation inpainting, a result ofthe non-segmentation inpainting, and information about a differencebetween the results of the segmentation inpainting and thenon-segmentation inpainting to the user, wherein setting the inpaintinggroups further comprises differently generating group sets based on anumber of inpainting groups, and wherein generating the group setscomprises generating a first group set in which the number of inpaintinggroups has been set to a first number of segmentations and generating asecond group set in which the number of inpainting groups has been setto a second number of segmentations different from the first number ofsegmentations.
 12. The inpainting method of claim 11, wherein thesegmentation inpainting comprises first segmentation inpainting adaptedto be performed on the first group set and second segmentationinpainting adapted to be performed on the second group set.
 13. Theinpainting method of claim 12, wherein the result of the segmentationinpainting comprises results of the first segmentation inpainting andthe second segmentation inpainting.
 14. The inpainting method of claim13, wherein providing the information about the difference comprisesfurther visually displaying information about a difference between theresults of the first segmentation inpainting and the second segmentationinpainting.
 15. The inpainting method of claim 14, wherein generatingthe group sets comprises additionally generating the second group set inwhich the number of inpainting groups is equal to or smaller than areference number of groups when a number of inpainting groups of thefirst group set exceeds the preset reference number of groups.
 16. Theinpainting method of claim 15, wherein generating the group setscomprises generating the second segmentation set by merging any two ormore of the inpainting groups of the first group set when the firstnumber of segmentations exceeds a preset reference number ofsegmentations.
 17. A non-transitory computer-readable storage mediumhaving stored thereon a program for executing an inpainting method usingsegmentation of a reference region, the method comprising: determiningan inpainting target region present within video image content and areference region composed of pixels present within a preset distancefrom boundary pixels of the inpainting target region; setting inpaintinggroups composed of respective pairs of segmented reference regionsobtained by segmenting the reference region using pixel values withinthe reference region and segmented target regions obtained by segmentingthe target region so as to correspond to the segmented reference region;performing segmentation inpainting adapted to perform inpainting on asegmented target region using the segmented reference region on aninpainting group basis, and non-segmentation inpainting adapted toperform inpainting on the target region using the reference region; andvisually providing a result of the segmentation inpainting a result ofthe non-segmentation inpainting, and information about a differencebetween the results of the segmentation inpainting and thenon-segmentation inpainting to the user, wherein setting the inpaintinggroups further comprises differently generating group sets based on anumber of inpainting groups, and wherein generating the group setscomprises generating a first group set in which the number of inpaintinggroups has been set to a first number of segmentations and generating asecond group set in which the number of inpainting groups has been setto a second number of segmentations different from the first number ofsegmentations.